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Maximum likelihood data sequence estimation, implemented by a 
dynamic programming algorithm known as the Viterbi algorithm (VA), 
is of considerable interest for data transmission in the presence of severe 
intersymbol interference and additive Gaussian noise. Unfortunately, the 
required number of receiver operations per data symbol is an exponential 
function of the duration of the channel impulse response, resulting in 
unacceptably large receiver complexity for high-speed PAM data trans- 
mission on many channels. 

We propose a linear prefilter to force the overall impulse response of 
the channel/ prefilter combination to approximate a desired truncated 
impulse response (DIR) of acceptably short duration. Given the duration 
of the DIR, the prefilter parameters and the DIR itself can be optimized 
adaptively to minimize the mean-square error between the output of the 
prefilter and the desired prefilter output, while constraining the energy in 
the DIR to be fixed. 

In this work we show that the minimum mean-square error can be 
expressed as the minimum eigenvalue of a certain channel-dependent 
matrix, and that the corresponding eigenvector represents the optimum 
DIR. An adaptive algorithm is developed and successfully tested. The 
simulations also show that the prefiltering scheme, used together with the 
VA for two different channel models, compares favorably in performance 
with another recently proposed prefiltering scheme. Limiting results for 
the case where the prefilter is considered to be of infinite length are obtained; 
it is shown that the optimum DIR of length two must be one of two possible 
impulse responses related to the duobinary impulse response. Finally we 
obtain limiting results for the case where the transmitting filter is optimized. 
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I. INTRODUCTION 

Forney 1 has recently proposed a receiver structure for a communi- 
cation system operating over a known time-dispersive channel with 
little loss in performance due to intersymbol interference by using 
maximum likelihood sequence estimation, or the Viterbi algorithm 
(VA). 2 This has resulted in much attention being given to practical 
methods of applying his results. Magee and Proakis 3 proposed the use 
of the VA directly in conjunction with a channel estimator. This 
approach can result in a receiver too complex for practical use because 
the complexity of the VA depends exponentially on the duration of the 
channel impulse response. 

In particular, if the impulse response of the channel has an effective 
duration of r seconds and if an L-level PAM system transmits l/T 
data symbols per second, the number of operations per received symbol 
is proportional to L T,T . For channels such as voiceband telephone 
channels, the bandwidth of which is used efficiently, typical values of 
t/T may be between about 20 and 200, making direct application of 
the VA infeasible. 

Thus, it seems clear that effective practical application of the VA 
or of related techniques involves a compromise between optimum 
performance and receiver complexity. The complexity-limiting ap- 
proach we take here is to use a linear prefilter at the receiver to 
"condition" the overall sampled impulse response seen by the VA so 
that it is significantly different from zero over only a small number of 
samples, and any remaining intersymbol interference is considered to 
be noise. Additional joint optimization of the transmitting filter is 
also treated, but would be much harder to implement in a real system. 

The simplest example of a prefilter is a linear equalizer, which yields 
an approximate overall impulse response of just one sample. Another 
example of prefiltering for a different purpose is the linear portion of a 
decision-feedback equalizer; in that case the initial sample of the de- 
sired overall impulse response is required to be large relative to the 
additive noise. 

In any application of prefiltering to approximate a desired impulse 
response (DIR), the DIR itself and the prefilter should be chosen to 
minimize the error due to noise and to the difference between the DIR 
and the actual impulse response that is achieved. The latter error re- 
sults from intersymbol interference components outside the interval 
accounted for by the DIR samples as well as from errors in approxi- 
mating the DIR inside the time-limited interval. This error could be 
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eliminated by using a zero-forcing criterion at the cost of additive 
noise. 

Qureshi and Newhall 4 have recently proposed a receiver incorporat- 
ing prefiltering with the VA. They use a mean-square error (MSE) 
criterion to force the overall response of the channel plus the linear 
equalizer to approximate a truncated version of the channel pulse re- 
sponse. In order to decode, the VA assumes this truncated response, 
resulting in much simplified processing. There is no effort made in 
Ref. 4 to optimize the desired truncated response. It is the purpose 
of this paper to see how this desired response can be chosen to minimize 
MSE and to show that this receiver structure can be made adaptive. 

In Section II we formulate the MSE-minimization problem, assum- 
ing a fixed number of samples in the DIR and in the impulse response 
of the prefilter. The minimum achievable MSE is the minimum eigen- 
value of a certain channel-dependent matrix. In Section III we indicate 
how the prefilter tap coefficients and the samples of the DIR can be 
determined adaptively by a gradient algorithm based on the MSE 
minimization. Section IV is a study of the limiting situation in which 
the tapped delay line prefilter consists of an infinite number of taps 
and it is preceded by a matched filter. Compact expressions for the 
prefilter impulse response, DIR, and minimum MSE are derived, 
which lend further insight. Section V describes the results of computer 
simulations of an adaptive prefilter/VA receiver structure, including 
comparison of the receiver with that described in Ref. 4 and with per- 
formance lower bounds. Section VI presents performance calculations 
for the prefilter/VA system, a decision-feedback equalizer, and a 
linear equalizer for a particular channel. Plots of minimum MSE 
versus bit rate for each of the three types of receiver structures are 
shown. Section VII considers asymptotic transmitter optimization. 

II. OPTIMIZATION OF THE RECEIVER 

The channel model and the preliminary receiver processing are 
shown in Fig. 1. The channel is modeled as a linear continuous filter 
with additive white Gaussian noise. It has been shown by Forney 1 
that the channel can then be followed by a matched filter, symbol 
rate sampler, and noise whitening filter with no loss of information. 
Alternatively, the reader may assume that the channel is band- 
limited and symbol rate sampling can be used with no information loss. 

Due to these considerations, the discrete-time model of Fig. 2 was 
adopted with the additional assumption that the channel pulse re- 
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Fig. 1 — Channel model and preliminary receiver processing. 




Fig. 2 — Discrete-time channel model. 



sponse is time-limited. The noise sequence {n k } is additive, uncor- 
related, and Gaussian with variance a 2 . Note that a discrete-time model 
with uncorrelated noise samples also results from the commonly used 
but nonoptimum expedient of passing the received signal through a 
flat Nyquist band-limiting filter prior to sampling. * 

The proposed receiver structure is shown in Fig. 3. The received 
sequence feeds a linear tapped delay line filter whose function is to 
shorten the overall impulse response length. The filter has L( = 2M + 1) 
taps which are chosen in the manner to be described later. The output 
of this filter feeds the Viterbi algorithm which detects the information 
sequence. 



* Although a white noise model was used throughout, the correlated noise case 
can be considered in a similar manner. 
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Fig. 3 — Receiver structure. 



The Viterbi algorithm makes decisions on the assumption that the 
DIR {q k }ti is the actual overall channel response. The value of LL 
(the length of the DIR) is much less than 2N + 1 (the length of the 
actual channel response). LL is chosen to make acceptable the com- 
plexity of the Viterbi algorithm while taking a small noise penalty 
in the linear preprocessing. * An error signal is formed by feeding the 
information sequence estimate through the tapped delay line repre- 
senting the desired channel response. This forms the desired truncated 
channel received sequence which is then compared with a delayed 
version of the actual linear prefilter output to form an error sequence. 
It is this error which is to be minimized since it represents a sum of 
the additive noise, and the difference between the desired and actual 
overall impulse responses. 

If the sampled channel impulse response, sequence of information 
symbols, and sequence of uncorrelated noise samples are represented 
respectively by {to},- _., {/»}£_., and {n f },"_., then the fcth dis- 

' Obviously if LL is allowed to be very large, the DIR can closely approximate a 
delayed version of the original channel impulse response, and there is no significant 
noise penalty, since the prefilter simply approximates a delay tine. 
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crete channel output is 

r* = 51 hilk-i + n k . 



(1) 



Then if the vectors P+ ■ (v-m, • • •, po, ■ • ■ Pit) where + indicates 
transpose and Q + = (q , • • •, ?ll) represent the tap coefficients of the 
prefilter and the DIR respectively, the error in the fcth interval is 



M LL 

e* = L Ptf'*-* - £ qih-i. 

I M l — o 



(2) 



In order to simplify the following equations, it is assumed that the 
information sequence is uncorrelated (J*/y = hi) and that the in- 
formation sequence estimate equals the information sequence. Sub- 
stituting (1) into (2) and averaging e\ we get 



where 



if = e 2 = P+AP + Q+Q - 2P+//Q, 

ho • ■ -hhh 



H = 



h—M ■ • • h- 



M+LL 



(3) 



(4) 



is an L X (LL + 1) matrix and A is an (L X L) channel covariance 
matrix with elements a,-,- = rWj- 

First, the error is minimized with respect to the prefilter by taking 
the gradient with respect to the taps {pi\ and setting it equal to zero. 
The taps {<?;} are constrained to be nonzero. 



and therefore 



H = 2AP - 2HQ = 



P opt = A-*HQ. 



(5) 
(6) 



The interpretation — thus far — is that some desired (and truncated) 
channel response is chosen ; and the linear prefilter taps are chosen to 
force the overall response to this with a minimum MSE. The question 
of what this desired response should be naturally arises. If a fixed 
length is assumed for this desired response, the desired response can 
be optimized in the sense of minimizing MSE. Substituting (6) into 
(3), one obtains 

? = Q+[7 - H+A- l H]Q, (7) 



where / is the identity matrix. 
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Since e 2 ^ 0, this is a positive definite quadratic form in Q which 
depends only upon Q and the channel characteristics. This can be 
minimized by choosing Q to be the eigenvector with the minimum 
eigenvalue of the matrix [/ - H+A~ l H~\. The constraint 

Q + Q = i (8) 

is necessary to avoid the trivial case of no MSE. The trivial case cor- 
responds, of course, to no transmission through the channel. 

It should be noted that when the MSE is minimized a reasonable 
definition of the signal-to-noise ratio (SNR) seen by the Viterbi 
algorithm is maximized. This is true because [q { ] is considered to be 
the effective channel pulse response, constrained to unit energy; the 
additive noise plus any residual intersymbol interference is the effec- 
tive noise seen by the algorithm. Since this noise is equal to the MSE 
which has been minimized, the SNR has been maximized. 

In summary, to minimize the MSE and thus maximize the SNR 
seen by the VA receiver, choose 

Qopt = eigenvector of [7 — i/ + A -1 //] corresponding to its 

minimum eigenvalue, (9) 

Popt = A-^Qopt, (10) 

and then 

e 2 min = min eigenvalue of [/ — H+A~ l H~\. (11) 

III. AN ADAPTIVE ALGORITHM FOR OPTIMUM RECEPTION 

In order to make the receiver structure practical, the procedure of 
choosing the {p,j and the {<?<} must be made adaptive since the channel 
pulse response will not usually be known prior to the start of trans- 
mission. An algorithm to choose the taps adaptively will now be 
described. 

Consider the conditions for the optimum operating point of this 
receiver to be reached. The condition that the gradient with respect 
to P be equal to zero is easily implemented by using the products of 
sampled values of quantities in the receiver as noisy estimates of the 
required cross correlations, assuming the data sequence is known or 
has been correctly estimated by the receiver. Thus, 

pcr+i) = per) _ A!e r R< r >, (12) 

where P (r) is the set of tap values at the rth iteration, A x is an adjust- 
ment parameter which controls accuracy and speed of convergence, 
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R (r) is a vector of the received samples contained in the linear pre- 
processing filter, and e T is the error in (2). P (r+1) is thus the new 
estimate of the {pi} taps, and when a steady state is reached a noisy 
unbiased estimate of these taps is obtained. Note that the value of P 
implicitly depends on the value of Q through the e T terms. 

The algorithm to obtain the Q taps is not so easily obtained. Consider 
the unconstrained gradient with respect to the Q vector. If a noisy 
estimate of the required cross correlation is used, then the recursion 
for the unconstrained gradient algorithm is 

Q(r+1) = Q(r) _|_ A 2 e r I (r) , (13) 

where I (r> is a vector of the information symbols contained in the 
channel reference filter. If (12) and (13) are followed without a con- 
straint at each iteration, then the trivial solution results. The algo- 
rithm is therefore modified so that the Q vector is renormalized at 
each step. That is, 

Q(H-1> = Q(r) + A 2 e r I (r) (14) 

QO«- ^ (15) 

V (Q(r+l)) + (Q(r+l)) 

By following the combined algorithm of (12), (14), and (15), a sta- 
tionary point in P will be reached, and the energy in Q will be con- 
strained to one. _ 

Now consider the noiseless unconstrained gradient of e 2 with respect 

to Q. Then 

-^ = 2Q - 2H+P- (16) 

Consider P to be in the neighborhood of the correct solution (6) with 
respect to Q (that is, P is adjusted more quickly than Q). Then (16) 
becomes 



% = 2Q - 2H+A~ 1 HQ- (17) 

Thus the gradient algorithm, in terms of the actual matrix quantities, 
becomes 

= q(d _ |Ai(2QW - 2H+A~ 1 HQ< r) ) 
= Aiff+A-yyQW + Q (r) (l - A 2 ) (18) 

and then Q( r + 1) is renormalized to form Q (r+1) . Now note that if 
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A 2 = 1 this corresponds exactly to the method of Vianello and Stodola 5 
for determining the maximum eigenvalue and corresponding eigen- 
vector of H+A~ l H. Since the maximum eigenvalue of H + A~ l H cor- 
responds to the minimum eigenvalue of (/ — H + A~ l H) this technique 
will converge to the minimum MSE. This method will fail only when 
the starting vector Q (1) is exactly orthogonal to the desired solution. 
Since the algorithm actually used (14)- (15) deals with noisy estimates 
rather than the exact expressions, the noise will prevent the case of 
the algorithm becoming stuck on a vector orthogonal to the solution. 
In the practical case it is not possible to choose A 2 to be one because 
when the noisy estimates are used the algorithm will amplify the 
noise and diverge. Actually, A 2 will be much smaller than one. Again 
looking at (18), one can see that a steady state is reached when Q 
becomes nonrotating with respect to the transformation. This occurs 
when Q is the maximum eigenvalue of H+A~ l H (i.e., the maximum 
eigenvalue will dominate as in the method of Vianello and Stodola). 
Thus, the unique solution for Q has been obtained. 

IV. LIMITING RESULTS 

We now study the limiting situation where the prefilter is allowed 
to be any general linear filter with impulse response p(t), while the 
desired impulse response {q m }m=o is still finite. In addition we assume 
that the additive noise on the channel is white, with double-sided 
power spectral density N /2. 

In this case we wish to minimize the mean square of the sampled 
error 

/•*> LL 

p{r)r{kT - r)dr - £ $,/*_, , (19) 

-» 1=0 

where 

r{kT - r) = £ h(kT - IT - T )h + n(kT - r) (20) 

is the received signal. 
The MSE is then 

i 1 = £ P P p(Ti)p(Ti)h(lT - n)h(lT - Ti)d Tl dT 2 

l= — X J— CO J —XI 

+ %r f X P(r) 2 dr - 2 £ Ql f p(r)h(lT - r)dr 

^ J-*> 1=0 J-X> 

+ !??• (21) 

1=0 



1550 THE BELL SYSTEM TECHNICAL JOURNAL, NOVEMBER 1973 

Using a simple calculus of variations argument to minimize MSE 
with respect to the prefilter impulse response p(t), we get the following 
integral equation denning the optimum p(t). 

%r Pit) = £ qMIT -t) - ± 8MIT - t) , (22) 

Z 1=0 l=-a> 

where 

Si = r p{r)h{lT - r)dr (23) 

J— oo 

is the overall sampled impulse response of the channel and prefilter. 
Note that we would hope f or ^ I ^ LL, Si « qi and for I < and 
I > LL, SittO. 

Equation (22) tells us that the optimum prefilter structure is a 
matched filter with impulse response h( — t), followed by an infinite- 
length tapped delay line whose tap gains {pi\ are given by (22) and 
(23). 

~Pi = qi~ t P*~-i, ( 24 a) 

where 

<f> m = r h(mT - r)h{-T)dr = 0_ m (24b) 

J— 00 

is the channel's sampled covariance function, and where we later 
require that [qi\ is nonzero only f or ^ I ^ LL. We remark that the 
development so far is analogous to that of Berger and Tufts 6 for the 
case LL = 0. Equation (24) may be solved in terms of z-transforms. 
Defining 

q(z) = L 9mZ m 

pW = E Pmz m 
4>{z) = £ 4>mz m 

m=— oo 

we can take z-transforms of both sides of (24) and solve for p(z). 

P (,)=-^V, (25) 

where we have used the fact that 0(z) = 4>{z~ l ) since the sequence 
[4>i] is symmetric about I = o. 
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Using (25) we get the z-transform of the autocovariance sequence 
of the {e k \, when the tap coefficients {p„}l„ are chosen to minimize 
the mean-squared error. Defining E m = e^e^ and E{z) = ££ = _ „ E m z m 
we have 

MM -%&>&£. (26) 

We now minimize e 2 = E with respect to the desired impulse response 
samples {q n }5L , under an appropriate energy constraint. Taking the 
inverse transform of E(z) we have 



E. 



= ^ / E(e-* T )dc 

4T J-r/T 



_N T fir g( g *»l)|l J 

ttL«— 7z nr.**' (27) 



where 



+(«-*") + ±L_° 



ff(c- r ) = g„ + g**" + ■ • • + q LL e*» LLT . 



Defining the LL + 1 dimensional vector Q+ = (g , q h ... ?LL ) we 
can rewrite (27) as a quadratic form 

E a = Q+RQ, (28 ) 

where R is a square matrix of dimension {LL + 1) whose i-jth element 
is 

Note that 0(e** r ) is the discrete Fourier transform of an autocovari- 
ance sequence, and hence is an even, real, positive function of «. Thus 
ra - fa is a real function of \i - j \ , and so R is a positive definite 
symmetric Toeplitz matrix. 

Minimization of E under the energy constraint |Q| 2 = 1 is then 
accomplished by making Q that normalized eigenvector of R cor- 
responding to its minimum eigenvalue. The matrix R is evidently the 
limiting case of the matrix / - H+A~ l H for the finite-tap receiver 
[displayed in expressions (7) through (11)]. Then 

3L = Amln(#). (30) 

To recapitulate, the minimum is taken over the set of tap-coefficients 
I p„j "__, and {g„}££ under the constraint £££ q\ = 1, 
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which can be expressed as 

T r vlT 
4~ I g(»**) «--!■ 

Z7T J-t/T 

Thus, from (27) we have the lower bound 



X mi n(fi) = E = e&m ^ ^ ^ ^-r- (31) 



2 



sup U(e>» T )+^) 



Now 4>{e iuT ) is the discrete Fourier transform of the sequence {</>„} 
defined in terms of the channel's impulse response by (24b). Thus, if 
the channel's transfer function is denoted by 



H(o>) = f K h(t)e-i» T dt , 

J to 



(32) 



The term <f>(e>" T ) can be interpreted as the channel's "folded" power 
spectrum. 7 

When LL + 1, the number of components in the desired impulse 
response {?«}££<,, is relatively small, say less than 10, the minimum 
eigenvalue and corresponding eigenvector of R can be evaluated with- 
out difficulty. For much longer values of LL, the lower bound (31) 
which is easily computed using (32) may be quite tight. A particular 
case of interest is where LL = 1. Then R has the form 

R=\ r ° ri l 

and 

eLm = Xmin(#) = min(r + r h r - n), 
where 

ro + ? , = ^/-_^f^ dw (33) 

and 

sin 2 — - 



= Ej± f 

2lT J-rlT 



0(e- r ) + ~ 



do,- (34) 
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The normalized eigenvectors (optimum (q 0) qi)) corresponding to the 
eigenvalues r + n and r - n are respectively (1/vZ, 1/vZ) and 
(1/V2, -1/V2). 

Thus, we have the curious result that the optimum desired impulse 
response of length two is one of only two possible forms, depending 
only on whether the channel's folded power spectrum is such that 
(33) or (34) is smaller. For example, if the channel's folded power 
spectrum has a single minimum near the band edge, a = w/T, the 
best choice for (q , qi ) would be (l/v2, l/v2) since cos 2 («T)/2 has a 
zero at the band edge. However, if the channel's folded power spectrum 
has a single minimum near zero frequency, the best choice for {q ,qi) 
would be (1/VZ, -l/vl), since sin 2 (uT)/2 is zero at a = o. These 
two cases are illustrated in Fig. 4. 

It is interesting to point out that the two possible optimum desired 
impulse responses (l/vl, 1/VZ) and (1/V2, -1/vZ) are reminiscent 
of duobinary and partial response impulse responses. 8 

V. PERFORMANCE OVER SIMULATED CHANNELS 

In order to observe performance obtainable from this receiver struc- 
ture, the arbitrary discrete time channels shown in Fig. 5 were used. 
Figure 6 shows the results of the simulations performed with the 
receiver developed here and that of Qureshi and Newhall on these 
channels. Underlined in Fig. 5 are the desired response used for the 
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Fig. 4 — Optimum desired impulse response of length two. 
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CHANNEL B 

Fig. 5 — Sampled channel impulse responses used in the simulation. 

Qureshi and Newhall receiver. As can be seen from the performance 
curves, the Qureshi and Newhall receiver performs about as well as 
our receiver for Channel A and much worse for Channel B. The 
difference in performance is presumably due to the use of different 
criteria to choose the desired response. In the case of Channel B, the 
channel passes virtually no dc, yet the DIR from truncating the 
channel response does pass dc. This causes considerable noise enhance- 
ment by the Qureshi and Newhall linear prefilter. 

Figure 6 also shows the matched filter lower bound, the lower bound 
on performance derived by Forney, 9 and a lower estimate which is 
used to predict actual optimum reference receiver performance. This 
lower estimate is obtained by computing the MSE and minimum 
coding distance of the DIR. Thus, it is assumed that the MSE is un- 
corrected and Gaussian in this approximation. 

™> K ^(\4Sh)' (35) 

where K is a constant depending on the error structure of the channel, 
and dmin is the minimum Euclidian distance between all possible 
pairs of noiseless sequences with differing first information symbols 
emerging from the prefilter. 9 This lower estimate is found without 
considering the fact that the noise is correlated. If a more accurate 
estimate of performance is desired, the results of Qureshi and Newhall 4 
can be used to consider the effects of noise correlation. 

The simulations were run with a 31-tap prefilter whose taps were 
adjusted with A = 0.001, and a 5-tap desired overall response length 
with the adjustment parameter equal to 0.01. In the case of the Qureshi 
and Newhall receiver, the prefilter was adjusted with A = 0.001 and 
the channel was estimated with a filter with adjustment parameter 
equal to 0.01.* As the curves show, the receiver structure given here 



* The performance loss due to the adjustment parameters has not been evaluated ; 
however, simulation results indicate that this loss is very small. 
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Fig. 6— Simulation results : (a) channel A ; (b) channel B. 



assures that a good choice of a DIR is made which is not always the 
same as the truncated channel impulse response. 

VI. COMPARISON WITH OTHER SYSTEMS — AN EXAMPLE 

Based on the results in Section II, performance calculations were 
made for baseband PAM transmission on the channel whose frequency 
response is shown in Fig. 7. The results shown in Fig. 8 were made with 
the following assumptions : 

(i) A matched filter preceded the receiver. 
(n) There was a 31-tap prefilter. 
(in) There was a 5-tap desired impulse response. 
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Fig. 7 — Channel amplitude characteristic. 



(iv) Although the noise may be correlated, at the input to the 
Viterbi algorithm, it has a negligible effect on performance. 

Of the assumptions made, only the one about the noise correlation 
might not be realistic. Work is currently being done to deal with the 
correlated noise problem. In any case, it is not expected that it would 
affect performance more than a few dB and it clearly would not affect 
the place in the performance curve at which the performance begins 
to degrade seriously. 

The curves representing the linear and decision-feedback equalizers, 
provided by J. Salz, show the MSE versus rate for additive white 
Gaussian noise with N„/2 = 0.0001. In the linear and decision-feed- 
back cases the MSE may be roughly related to performance in terms 
of probability of error. * 6 ' 10 The curve for the prefilter/VA combination, 
labeled "VA equalizer," is a plot of (MSE/dSuJ versus rate, where 
d min is the minimum distance for the DIR. This is done because the 
attainable system performance is not determined by MSE alone, but 
rather by MSE/dL, as in expression (35). Direct minimization of this 
ratio by analytical or numerical means has not been accomplished. 
Note however that the minimum value d mia can attain (over all 



*The analysis for decision-feedback equalization ignores the effect of decision 
errors on the MSE. 
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Fig. 8 — Indication of attainable performance for three receiver structures. 



channels with equal-energy impulse responses) is limited by the dura- 
tion of the DIR which is chosen. 11 

As can be seen from the curves, this receiver structure can be ex- 
pected to perform well while using only binary signaling over a much 
greater range of transmitted data rates than the linear and decision- 
feedback receivers. This result occurred despite the fact that the 
linear and decision-feedback computations were made for infinite 
filters while the prefilter was finite. It is the more relaxed criterion for 
our system compared to the decision-feedback criterion which results 
in lower MSE and thus better performance. Nevertheless the results 
are considered preliminary until a better understanding of the effect 
of noise correlation is achieved. 

VII. TRANSMITTER OPTIMIZATION 

The "channel's" frequency response H{u>) actually includes the 
transmitting filter, i.e., 

H(u) = C(«)G(co), (36) 

where C(w) is the frequency response of the transmission channel 
alone, and G(u) is the frequency response of the transmitting filter, 
which we have hitherto assumed fixed. In a practical data communi- 
cation system, a "reasonable" transmitting filter would likely be 
fixed to avoid having to provide an extra feedback channel for adjust- 
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ing the transmitter parameters, and because of the complexity of the 
transmitter optimization argument itself for general channels. 6 

Nevertheless, the performance attainable with transmitter optimi- 
zation is of theoretical interest. In this section we obtain expressions 
for the optimum transmitter filter G(o)) and the resulting minimum 
MSE under a transmitted power constraint. For simplicity, we assume 
a "well behaved" channel C(«) for which | C(«) [ is monotone decreas- 
ing, and for which | C(w) | /N„ is sufficiently large in the range { -tt/T, 
ir/T) that the optimum transmitter uses the entire Nyquist band 
| cd I ^ tt/T. Treatment of more general channel characteristics is more 
complicated, but can be carried out as in Ref. 6. 

The minimum MSE for a fixed transmitting filter (?(co) and DIR {qi\ 
is given by eq. (27) and by the channel's folded power spectrum, 
which from (32) and (36) can be written 

*^-f._ll c (— WK— t)|'- (37) 

The constraint that the transmitted power be fixed at P r can be 
written 

*C?K-t)|'*-*- < (38 > 

A necessary condition for minimizing the MSE, given by (27), 
subject to the power constraint (38), is obtained using a simple vari- 
ational argument: for -tt/T ^ co ^ ir/T and every integer m, either 
G(u - 2wm/T) = or G(a> - 2vm/T) ^ and 

?.£.rv - ~"3 r J| \ G \ a ~~r)\ + ^~ 

= Xk(e^)||c( w -^)|, (39) 

where X is a Lagrange multiplier whose value will be determined from 
the constraint (38). Furthermore, for any at such that C(w) = 0, 
G(eo) = 0. 

For any frequency u, there will be only one integer m for which 
G(u — 2rm/T) ^ 0, since the left-hand side of (39) does not depend 
on m and the right-hand side does. Indeed, if |C(w)| is monotone 
decreasing, then best use is made of the transmitter power if G(a>) =0 
for | Co | > tt/T. Thus we can rewrite (39) as 

i|C(«)|MG( w )| a +^ = X|g(^ r )||C(«)| 

for C(«) 5^0 and |«| ^ it- (40) 
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For simplicity, we assume that P T and the ratio \C(u)\/N are 
sufficiently large that (40) can be satisfied for all |w| < t/T. [Other- 
wise G(u) would be zero 6 beyond a certain frequency w < v/T.~] 
Then the amplitude frequency response of the optimum transmitting 
filter is given by 

y|G opt (o,)| - , C(w) , -2|CRT 2 for|tt| ^f 

= for to >|- (41) 

The Lagrange multiplier X is determined by (41) and the power 
constraint (38). Substitution of the expression for JG 0P t| into expres- 
sion (27) for the MSE gives 

MSE "4^-y_ r/r -[cRr dM ' (41a) 

where 

x j i-wT CM 2 ,.,. x 



/: 



/T 



|C( W ) 



It is interesting to look now at the frequency response of the optimum 
receiver prefilter 

i |Pj(«)|«-^ |C(co)| 2 |G op t(co)| 2 |p(a*")l», (42) 

corresponding to the optimum transmitter filter. The left side of (42) 
follows from the cascade of the channel and transmitter and the appro- 
priate matched filter, followed by the discrete filter {p t \. Substitution 
of expressions (41) for |G OP t(«)| and (25) for p(e juT ) results in 



JilPitol"-^ 



for I a, I g ^- (43) 



|C(«)| 2|C(o,)| 2 J ' UI |W| = 7 1 

Thus the transmitting filter and receiving prefilter frequency re- 
sponses are identical in the Nyquist band except for constant factors 
(clearly, the transmitting and receiving filters' phase characteristics 
can be chosen arbitrarily). This equal sharing of the filtering load 
between the transmitter and receiver is a well-known result for 
optimum linear communication systems (see pp. 118-121 of Ref. 7). 

It is also of interest to evaluate the power spectrum of the error 
sequence that the Viterbi algorithm assumes to be additive uncorre- 
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lated Gaussian noise samples. From expressions (26) and (40), this is 
given by 

Me } 2 X|fl(«*")||C(«)| 

= 2xWT for|w| = r' (44) 

Thus, the extent that the amplitude frequency response of the 
chosen DIR approximates that of the channel in the Nyquist band 
determines how close the power spectrum E(e i,tT ) is to being flat, 
and hence, to what extent successive errors are uncorrelated. 

From (41a) and (41b) we obtain an expression for the MSE for a 
given DIR after the transmitting and receiving filters have been 
optimized. 

MSE = Ar „,,. i , (45a) 



where 



/TIT 



No f rlT 1 
2 J- r ,T | <7(<o) | 2 

*IT \q(e'" T )\ 



C(«) 



[LL LL -\\ 

- , , -°""i > rM | i ^' (45b) 

Minimization of the MSE expression with respect to the DIR Q 
under the constraint | Q | 2 = 1 is then equivalent to minimization of 
a(Q) under this constraint. Necessary conditions for the optimum 
Q + = (Qo, ■ ■ ■ Qll) are then 

fiqi = E qm P i- m (Q), I = 0, • • • LL, (46a) 

m=0 

where 

/T r-n^LZ t*^ (46b) 

- /r |C(co)| E Z? i9 i«-^ 

L i=0 A=0 J 

and where n is a Lagrange multiplier. Again the optimum DIR Q 
is the solution of an eigenvalue problem, this time nonlinear. It is easy 
to verify that the optimum DIR of length two is again either of the 
two "duobinary" impulse responses shown in Fig. 4. In that case the 
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minimum achievable MSE is given by (45a) and (45b) as 



N T [ [*it [1 ± cos ttfT]» I* 



ir [j- rfT 1C(«), 

JV„ rx/r i " . (47) 



MSE = 

P ,N /•"* i 



the + or — being chosen to minimize (47). 



VIII. CONCLUSIONS 



We have presented a scheme of linear prefiltering to optimally 
"condition" the impulse response of a channel to approximate an 
impulse response of limited duration for which maximum likelihood 
estimation of the data sequence is implementable in practice. This 
scheme in conjunction with the VA can be adaptive, to deal with un- 
known or slowly time-varying channels. In the simulations its per- 
formance compared favorably with the similarly motivated scheme of 
Ref. 4. 

The optimization criterion we used— minimization of the MSE with 
respect to the prefilter taps and the DIR, with the energy, duration, 
and relative delay of the DIR being fixed — is admittedly somewhat 
ad hoc. If the sequence of errors fe,) emerging from the prefilter is still 
assumed to be stationary Gaussian, with zero mean and covariance 
( E m } , then it can be shown that the error rate of a VA which assumes 
correlated noise is minimized if a certain weighted minimum distance 
is maximized, namely 

min d + A -1 d, 

where the s' is the set of all possible vectors representing error events 
and A is a covariance matrix whose dimension equals that of 
d{Aij = E\i_j\\. The above quantity is clearly difficult to maximize, 
and even if it could be done, the non-Gaussianness of the error se- 
quence would render the solution suspect. 

Nevertheless, the performance estimates for the sample channel 
reported in Section VI make the use of the VA is conjunction with 
prefiltering appear attractive for high-speed data transmission relative 
to other schemes. Further studies should be done on the correlatedness 
of the error sequence and the minimum distance properties of the 
desired impulse responses. * 



* S. Fredricsson presented a paper dealing with this subject at the International 
symposium on Information Theory, Israel, June 1973. 
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